A.M.; “Dynamic Augmented Reality for Sensory Substitution

Abstract — Teleoperated robot-assisted surgical systems are increasingly being used in minimally invasive surgery because they provide surgeons with improved precision, dexterity, and visualization. The addition of haptic (force and/or tactile) feedback has been proposed as a way to further enhance the performance of these systems. However, due to limitations in sensing and control technologies, implementing direct haptic feedback to the surgeon’s hands remains impractical for clinical application. Thus, the goals of this work are to develop an intu-itive augmented reality system for feedback of force information through sensory substitution, and to evaluate its performance in a knot tying task. The augmented reality system consists of force-sensing robotic instruments, a kinematic tool tracker, and a graphic display that overlays a visual representation of force levels on top of the instrument tips. The system is integrated with the da Vinci Surgical System (Intuitive Surgical, Inc.) and tested by several users in a phantom knot tying task. Results indicate that the visual force feedback system decreases the number of broken sutures, decreases the number of loose knots, and results in more consistent application of forces. I